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ABSTRACT 
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(for 3,386 papers) and an analytical scoring method were used in the 
PNA. Results show thatt (1) SSAT-I results agree with those from 
analytically scored production writing only when students have fairly 
high levels of overall writing skill; (2) scores representing the 
same skills for below-average students tend to be lower on the SSAT-I 
than on the analytically scored papers; and (3) SSAT-I results tend 
to penalize students with lower levels of overall writing skills. It 
appears that use of the writing portion of the SSAT-I as the 
criterion for labeling schools deficient is, at best, debatable. 
Examples of holistically scored students' essays from the 1986 PNA 
and results of analytical scoring of the PNA are appended. (TJH) 
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EXECUTIVE SUMMARY 



This study compares the results of an Indirect and a direct measure of writing ability: the 
Florida State Student Assessment Test, Part I (SSAT-I), and the Production Writing Assessment, 
both administered to Dade County tenth graders in the spring of 1987. The purpose of this study 
is to determine similarities and/or differences between results of these two instruments. 
Findings will be disseminated to the district school board and local schools. 

Description 

The SSAT-I is an untimed test containing a section indirectly measuring writing ability through 
multiple-choice items addressing specific skills such as punctuation, capitalization, spelling, 
and grammar. The Production Writing Ass'^ssment is a direct measurement of writing 
perfonnance requiring students to wriie an essay on a given topic within one cl&ss period. 

For this report, two procedures were used in the Production Writing Assessment. One involved 
holistic scoring, the other analytical scoring. Holistic scores for the 3386 papers in the sample 
provided an overview of students* general writing ability. The randomly selected essays were 
independently rated by professional readers on a scale of 1 (low) to 4 (high). Final scores were 
ihe sum of the two scores. Second, randomh/ selected subsets of each holistic score group in the 
sample were independently and analytically scored to obtain skill-specific information 
regarding students* ability to apply the SSAT-I skills. 

To estimate the reliability/percent agreement of the two analytical raters, the analytical scores 
for two skills from papers of the lowest holistic score group (2-3) were compared. For the two 
skills,the percents of agreement between the two raters' scores were 99% ajid 97%. Thus it is 
apparent that any difference between the SSAT-I and the analytically scored results is not likely 
due to a lack of consistency/reliability in the analytical scoring procedures. Results of the 
SSAT-I and the analytical scoring were sorted by the four holistic writing score groups (2-3, 
4, 5-C and 7-8) and then compared to determine if the two measurements of the same skills 
would yield similar results. 

Eindiogs 

Overall, students perform less well on the SSAT-I items measuring particular skills than on 
these same skills in actual writing measured analytically. This difference is greatest among 
students with low holistic writing scores ,2-3). The SSAT writing results are of limited use in 
diagnosing, on a skill-by-skill basis, the type and frequency of errors a betow-average student 
will make in actual writing. 

More specirmally, the findings indicate the following: 

( 1 ) Results of the SSAT-I agree reasonably closely with those from analytically scored 
production writing only when the students have fairly high levels of overall writing 
skill. 

( 2 ^ For below-average students, scores representing the same skills tend to be lower on 
the SSAT-I than on the analytically scored papers. 
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( ? ) The SSAT-I results tend to penalize students with lower levels of overall writing skill. 
Moreover, the SSAT-I multiple-choice format may even lead to a misdiagnosis, 
particularly in subject-verb agreement and irregular verb usage, of these students* 
major problems. 

( 4 ) Such diagnostic inaccuracies and/or insufficiencies and the deficiency label placed on 
schools when SSAT-I scores in a skill area fall bek)w eighty percent passing may lead 
to incorrectly focused instruction. The possble result can be to harni rather than help 
the student, and to misinform the instructor and/or the curriculum dsigner. Quite 
simply, these results appear (to us) to call into question the usefulness of the 
diagnostk: infonnation arising from SSAT-I writing results, at least at grade 10. 

( 5 ) Ftorida law requires that schools with SSAT-I results befow a certain nrJnimum be 
labeled 'deficient." Using the writing portion of the SSAT-I as the criteria for labeling 
schools deficient is at best debatable because the SSAT appec s to be a questk>nable 
indk^ator of the actual writing ability of k)w-level students. Further, because of the 
nature of writing, exclusive use of the multiple-choice format writing portton does 
not appear to be an adequate assessment for the measurement of writing skills. 

Conclusions 

From a somewhat speculative point of view, it is thought that the higher analyttoal scores arise, 
at least in part, from students' knowing they do not know, and therefore avokJing words or 
phrases of whch they are uncertain, and substituting structures less likely to be wrong. 

In any case, assessing writing skills indirectly through objective measures yiekJs one set of 
results while direct assessment by means of production writing may yieki another. That each 
has its advantages and disadvantages suggests that the exclusive use of one or the other may lead 
to misinformation regarding the level of students' writing skills. 
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A COMPARISON OF 1987 RESULTS OF SSAT-I WRITING 
AND PRODUCTION WRITING ASSESSMENT 



PURPOSE OF STUDY 



The purpose of this study Is to determine similarities and/or differences 
between results of the March 1987 SSAT-I Writing and the February 1Q87 
Production Writing Assessment, both administered to Dade County tenth g7adi?s 
liM^^^a^ /"r'"P* ""'•t^^n whether student perfoSnce^o^^^ 

s\'idiJMciM^ '''''' - ^ <'^-* nt 

DESCRIPTION OF ASSESSMENTS UNDER COMPARISON 

lonJ^T^^l^fiL'^f^Af^'^^i' ''f^ ^ instrument that 

S: f** section indirectly measuring writing ability through multiDle- 

iJ^pSni'tStiirVaVi"^^^^^^^^^^ 'i''^'' A«ong\he7kiTls"'i;cl5d^ 

are punctuation, capitalization, spelling, and grammar. (Refer to AoDendix A 

(Sr'tacf^J?^ ^''^ SS'^T-I ^^^t^ng skills addressed on the March 

1987 test.) Scoring is on a pass-fail basis determined by the criteria of aS 
correct (four of five items) for each of nine skills Vnd 7.4 corUa lseSen Sf 
ten) for the tenth (spelling). Results of the test are usld for the ?MtiJl 
"f5f5f"^°/ st"<'ent5' basic skills (to diagnose whethVr rlmediati^^^^^ 
needed) and for the determination of schiols' deficient/nondeficieTt statu^^ 

.^''^''^'^J'^ administered in an attempt to gain an 
pJSJr« H °" °/ state-mandated Writing Enhancement 

t^TIT; onp "easurement of writing performance requiring students 

to write one essay as a response to a given topic (prompt) within one ria« 
period. Test directions suggested activities fT nrewHt^^^^^^ 
revising/proofreading, and offered advice for budgeting tilie, P"*"^^*^"^ and 

ih'\ti^''l!!J!J!!* ""ST®"! '^?°'^' procedures were used in the Production 
Writ ng Assessment. One involved holistic scoring; the other analvtirai 

gen%'rJ?-wJ il:MtJ''V%^?°'^^"5 "5*"^' provideS^n ove^li^'of U Je 
lS «rh ?cc?J"*^- In this procedure, professional readers holistically 
Judged each essay's overall effectiveness regarding focus oraani«tion 
sentence structure, vocabulary, and mechanics. Randomly selected Issavsie^ 

cteTt^re^h^'fl'if^Th."^^^^^^^^ "J °" MlJw)"io"i1h? hTM a7 

scores were tne sum of the two readers' scores, from 2 to a UnnanHiv r 

T^V^' ^ """'l" 9""" """op*' by oldTcounty fir use to the 

scoring coapany, and exaq>1es of studants' essays.) " 

»"»')^'"' scoring nethod was employed to obtain sklll-sDetlfic 

rlild-fy 's'^Vii 's"„bs\' I oTe»«l'f^: Ja°ch' of^l'^H ^'r*""' 
groups. Specific sTtlls flouted a^' "ll' th 1 III? 

^.X1'1zIt'?o7^^^^X';n^v^as:a^"^■' ~"'»: 

Results of the SSAT-I and the analytical scoring were sorted by four holistic 



OUTCOMES EXPECTED OF THE STUDY 



Student performance on the objective test (SSAT) was expected to be better 
than, or at least equivalent to, performance on the analytically scored 
Production Writing Assessment (PWA) essays for the following reasons: 

1. Multiple-choice recognition items are generally thought to be easier 
than production writing items. « easier 

2. Each section of an objective test focuses on a single specific skill. 

3. The SSAT is untimed, freeing students from the pressure of time 
constraints. The PWA was administered in one class period, requiring 
students to read an" "unfamiliar- prompt, to plan and write a rSponsi 
to it, and to revise the essay that resulted. respunse 

** JacSri^S^^l" reported in the Journal for 

Research In Mathp..y,^^^^^ Hlli:ilt1lTn (1987, Vol. 18, No. 3 , 197-214) 
S??S;:lnT' "^^.""liJPle-choice and open-response ' test formats shii 
different results. When answers for both formats were scored right or 

S"°f3 ;'l^*'°u"* P**"*^*^ the multiple-choice fomat general ll 

yielded higher percentage correct results. generally 



PnOCEDURES OF THE STUDY 

I*'%^oliSVik IV^^ «««ur«iient of writing ability was conducted by Fred 
r^lat^ve vanHi^v^lf ''^i^^^'K* investigate the 

rtlillifJn ^^^^^^^^ approaches to the measurement of English 

^"^^l^^- findings indicate that multiple-choice items tend to 

Sf 's'^ski'm^fj'' °/Jk^"^?'^' '^'^^'^ bu^\hat the best ""s^^ 

witing. combination of multiple-choice items anH production 



EJS^?n^J?2„*Rn?Ji'*'*?Q'i;«\" Th; Measurement of Writing Ahn^^Y (College Entrance 
siSlXJi n.a5;ir^' 1966), while objective measures if wr tiig skills may be 
somewhat predictive of levels of students' actual writina D&rforTt»lro f^l 

P;"^^'*"'" * direct ^asur^ of lru[ng ab "ty^^ 

Ts^^JfM"^^""*" ""^ questions or interlinear exercises n comb natiiS 
with objective questions. In an interlinear exercise, a student is S ?Jd t! 

prssaqeTand\he„"'tr ^'^'"'^ ^'i t^n plSr^ri^Uten 

dlfici^nciJ \fuZfl° ^"PPly "O'-e appropriate forms to correct those 
aenciencles. Students write their corrections between the linp« nf *tZ 
passage hence the term "interlinear-. This is not to be confused with ?he 
AcMVveMsV.'''"'^ ''''' example"" the Su'Sfo^d 
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In the current study, the holistic scoring method was employed to gain an 
overall impress on of students' writing ability However. iJ order not to 
overlook specific characteristics of good writing, sn analytical scoring 

2ilf.«:^«„''i'f''^S!". 3rts teachers and othe? 

professionals. It is a five-point set of standards addressing focus, 
organization, sentence structure, vocabulary, and mechanics. This rubric was 
used to guide the identification of essays as "anchor" papers, those essays 
?JJHiHn^^^^^?cM"'' tyP^^y each score. This procedure is contrary to 
traditional holistic scoring. That is, in a true holistic procedure, raters 
illfJ^"" 2^ standards guiding their score assignments. 

rating is based only on an overall impression of the paper from 
excellent to poor, relative to all other papers in the study. 

nJJr^n ' w'Sc^nn g"^<'e to provide a score that reflects only an 

^" * combination of the analytical and holistic 
5 J* requ res readers to Judge an essay's merits based on pre- 
JJi*'LV±^K*"*L^*^"U^''^i^r^*' ^"'^l^ ""^^"9 them to provide a score for 
iSfiw^t?! *'*?J'*^^ impression only. The committee used the 

analytical-based rubric to identify papers that typified the characteristics 

;L JS^'^r. J.r.''n/'LV TP?'?:*^- "^"^ "^'9"«<' t° anchor pape? 

was the consensus of at least three committee members. Two anchor papers for 

each score were sent to the North Carolina scoring facility, along with the 

MnJ"?;;!?; SdalL"„'?r'S .'"^ ^^ri''"^' raters'^Ure "o maintJ n 

cl* iu 1?-**'®^'^ Judgements. Raters' Judgements were then subject to standards 
set by language arts professionals, and were not based on a paper's relative 
merits. The result was that a modified-holistic scoring method was em oy id 

the elements rf the analytical and holistic 5cS?ing 
methods, provides results that are less subject to the "relativi?ti"" 
criticism that pure holistic scoring might elicit. relativisti. 

The holistic scores used herein provided a picture of students' overall 
T.^Vil JiV^rl «'«',<'^vided their essays into seven score groups from ul {Z) 
rJ,lt lo^*/"^"'*?" simplify data analysis, these seven sets of sc?res were 
reduced to four: those with holistic scores of 2-3 (107 students^ 4 n73q 
students), 5-6 (1218 students), and 7-8 (322 students) Then, because hoUsUc 
J^I't'ho °f Aill-specific iiformation, random subse?S of pijers 

JKTirLrss^errtrfM^ "^-^^^ ''''' ^^^^^ 

l^re J^oJd."^ nf^\ low-to- .ligh holistic writing 

llZlf^f hSu ?! J'^* *' Passing/failing percentage's were 

5°^!'*^^ Sroup for each of the ten writing^kills asseHed 

i?!S^. JI'Ik 2^ '•'^"^ <8 18' ^^''^'-^te headings for grSSJs 

Mi^l^^Wy'iVi°n^^^^ ^^'^^'^^ 

JddJ.«led"bv S°/\'!;V?'.n5"^' *""^t^"g skills 

JnH Vfi^^S? nf^'^^Z^u*"^ individually. According to Spandel 

iwi, analytical scoring is appropriate if the intended outcome is to measure 
and report students' ability to deal with specific conventions of wHtiSS 
such as punctuation, capitalization, spelling, and grammar. writing. 
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Ill IJJ/'/^^l®®*" "^'^ writing score group were randomly drawn 

iLn"?V'i^'%''?^ corroborate the initial findings in the lowest sc^e 
JSSS il:*;V fol'^^^c scores of 2-3). ten additional papers were randomly 
t^Uin^I p.H^.r"? "'^"9 ^^'^ analytical method of scoring Sss^s 

eSl;^"on' n^'e^'^/epafSe'sSin' ''''''''''' independently ra?ed ea?h 

IrJiir estimate the reliability/percent agreement of the two analytical 
"fl^"^?:. ^''^ twenty-fivft papers from the lowest holistic scoring group were 
SJ^h ?h«JT/ '•«P7"r*s..' *"»lys^s because it is the^roup f5J 

Siu5nnM?„J! disagreement between the ,>SAT and the Pfoduaion 

Writing results (see "Resjlts of the Study" later p-esented). and it is the 
group for which the students' production wfiting error rates werfthe hi ahest 
For this group, two of the nine assessed skills were randomly seleSLl tfSak; 
the comparisons between the two independent scorers' results The two skills 
Ifflrl^i were spelling and subject/verb agreement. Because of the tiiS^ and 
effort-involved in calculating scorers' agreement, thirteen of the twentV- five 
Mapers were randomly selected to assess agreement in the skill of sSelliw 

orsu«":;ty:^Je;\,-^ ™' 

were subsequently converted to percentages indicating the number of misspell S 
words per hundred On average, the difference between the error ratJsfSr the 
iJJdfTrnM*Sli'^^5"J^^"' P^'"""*; that is. f^ 99 out of 100 

?l^h%i;'re'l?an99,''^"' correctly/incorrectly spelled, they agreed 22 

Regarding subject/verb agreement, the second randomly selected skill the 
percentages of agreement were slightly lower, but still high In this 

feTuU;'\^;o«'Vh??Jr*^' P^^""* °^ between'The^tt scorJf ' 

results, across the twelve papers, was 97%, 

?*';/^J<*^''gV it is apparent that the scoring done by the two 

b2?S tli h:"^';^.*'"ii^"^^" ^l^*'^!^' that the different 

between the SSAT and the analytically scored results cannot be due tc a lark 
of consistancy/reliability In the analytical scoring procedures 

JlIp°''<iYT ^J-H^^J^P * calculating a common type of error rate for 

l^ls/Au JntpH^. analytical scores of the production writing, the SSAT 
pass/fail criteria were applied to the analytical scores That is 
Em^h"*- '""/"^ was required to pass each skill except^^^^^^^^ G 58)^fo? 

ln^.uT!rXr^^^ the SSAT^ However, in tZ 

analytic scoring of the essays, the criteria for spelling mastery was stricter 

s5;;e'ns\u''tecausrar\h':r '''' ^'f ''"^r (n tMi^t" t' ? Sg 
score useiT. because as those scores increased, so did exoectatirn* nf 

sJllSllh*"*'^""""*"?;.*,"^ <2) the consideration that. °n producfion^ w^^^^^^ 
ti^rinn^JS ""[f *° ^'^^'^^ t^ey know how U yil As a result ?f 

these considerations, incorrectly spelled words could not exceed seven five 

r^Jictivtly''''' ''''''' "^^^'"9 ''''' 9r?ups Ti'TsT J l B 



RESULTS OF THE STUDY 



S?S? A' °" Pf^e^ illustrates, students perform less well on the 

SSAT itens measuring particular skills than on these same skills in actual 
writ ng. Moreover, this difference is greatest when students have low holistic 
writing scores (2-3). These students made ten percent or higher analytical 
scores on five of the nine skills (i.e.. those measured on the multiple-choice 
ItliAZt^^S^^^^ analytically on the written passages), and they made 
significantly lower scores on none of the nine skills. Only for the qrouo 
earning the highest holistic scores (7-8) did the results of the two 
procedures yield essentially equivalent results. Even here, however, there was 
some disagreejnent on two of the nine skills; in one case. SSAT scores were 
higher and in the other, the analytical results were higher! 

Somewhat more specifically, in the three skills focusing on subject-verb 
agreement, use of irregular verbs, and spelling words with suffixes, the SSAT 
results tend to differ substantially for the low holistic score group. SSAT 
results here were twenty or more percentage points lower than the re'^ults from 
the analytical scoring. A more detailed discussion for the low holistic score 
group follows. 

yAT-I, Skin A 10 - HAKF SUBJECTS ANO vfQK ^r,ppp 

JSIt k^'i!. the lowest holistic* score group passed this 

SSAT skill by identifying the correct form of subject or /erb in at least 
.our of five given items, but 56% of the essays scored analytically 
contained no errors at all in this skill. Another 36% of the essays had no 
more than twenty -ercent errors; therefore. 92% of the 2-3 students 

^f?^L?p^.:t*S^ ^ ^^^^^ ^" writing practice, a 

difference of fifty-four percentage points over the SSAT results. 

fu^Vib y^V^ APPROPRIATE FORMS OF COMMON TRRFfii|| flR yfrpp^ 

holistic score group. Of the twenty-five essays studied from this group. 

^Sth^or^ttTL"*?^ ^" irregular verbsrSSd 

another 24% had no more than twenty percent errors. Thus. 76% of these 

?JS*j5'thl'ssJT, successfully in actual writing. 33% mo?l 

JJ^"//''^! GENFRALIZATTflMS FhP A DDING COMMnN ^MFFTYf C 

J" i*"®. 66% Of the Students in the lowest holistic score group passed 

af iJ" 1 ; S^L^^^n^PP^^ "? '•^^^ *° the analytical sJSres? 

all in this group "passed" in actual writing. 

£ gX"/s^r;te"T'°" " "-P'H.on for the 

Considering the overall results for the study, there was only low- to-moderate 

fpearSnl t'i^^Jt '''^'^r'"'' ^"^ 'sti? scores''?{e 

;?^SInf;L?tr?l?l**",/i! ""''ly 3200 students. On average, 

students with low holistic scores tended also to have lower SSAT multiole- 
cho ce scores, but the skill-by-skill comparisons provided herein ?i5 ?a?e 
quite strongly that ski 11 -specific SSAT writing results are of limited use «n 
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PERCENT PASSING WRITING SKILLS 
IN SSAT-I AND PRODUCTION WRITING ASSESSMENT 



SKILLS 



A 10 

SubJ-verb 
Agreement 

A 11 
Irregular 
Verbs 

B 20 
Organize 
Topic 

G 58 
Spelling 

G 61 
Spelling 
Suffixes 

H 68 
Comma In 
Dir. Add. 

H 69 
Apostrophe 
Poss. Nouns 

I 74 
Caps In 
Titles 

I 75 
Caps on 
N/AdJ 

# of randomly 
drawn pap s 



HOLISTIC WRITING SCORE GROUPS 



Low 
2-3 

SSAT PWA 

38 ** 92 

43 ** 76 

65 64 

68 ** 80 

66 ** 100 



56 (not 
used) 



32 29 
(28% used) 



16 (not 
used) 



44 ** 59 
25 



Average 
4 

SSAT PWA 
60 ** 100 

75 ** 87 

81 ** 100 

85 87 
81 ** 100 



74 (not 
used) 



47 * 29 
(47% used) 

37 (not 
used) 

73 * 50 

15 



High Average 
5-6 

SSAT PWA 

81 ** 100 



89 93 



93 93 



95 100 



91 (not 
used) 



High 
7-8 

SSAT PWA 

93 100 

97 100 

97 100 

99 100 



90 ** 100 98 100 



95 (not 
used) 



61 63 73 * 33 

(53% used) (60% used) 

53 ** 80 73 **100 

(33% used) (27% used) 



87 * 67 



15 



92 85 
(87% used) 

15 



**SSAT 10%+ < Production Writing 
*SSAT 10%+ > Production Writing 



2;1 



5 
1 



4 

2 



^ h& 



3 
1 



Usage rates in PWA below 25% are shown as "not used" 



Total of 9 skills 
and levels 
13 
4 
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iJllAJ^xi*'! VP® ®^ * student will make In actual 

writing. That is, the SSAT writing results can hardly be considered 
diagnostic, on a ski 11 -by-ski 11 basis, for below-average students. 

From a measurement point of view, the weakness in SSAT "diagnoses" is quite 
understandable, even predictable, and this for at least two reasons. First. 
dl^SLl?. ^^T^^k'"'* ITJ?*" <'""*^ons on any skill for the results, i.e., th4 
diagnosis, tc be reliable. Secondly, the questions used to "measure" a 
Kiiiow more than occasionally appear to have questionable content 

At tiroes this is due to the nature of the skill when placed into a 
multiple-choice assessment format, e.g., in Skills G 58 and G 61, where 

fi?2rVr«m'P,'VL"tVn identifying the correctly spelled 

forms from a controlled vocabulary list, rather than by correctly spelling 
words used in composition. At other times, it appear? that one or mori 
specific questions are at best rat' ^r obtuse measures of the skill! 
particularly at a mini mum- performance it. el. To illustrate, in Skill H 69 
students are to identify a correct possessive form using an apostrophe and "s" 
for nouns not ending in "s", as in. for example, "wharf's". 

For still another reason, the two sets of scores (one derived from analysis of 

*'^^f*!^1//'"5T" ^i"^ °**'®'' skill-specific multiple-choice iteJs mSy 
well differ. In actual writing, the student may know he does not iS^ow. !?d 
therefore avoid using the word or phrase in question, substitutina in its 

S ?"nrimol''i5%f " •^"°*'k '''' knowledg^/skil°^'s'st'eVy'U"o^ta^^^ 
ZlUal* ^^ "n never be measured in a multiple-choice format - a 

C]m:ti;r?;'ma\ts?"s?° ^^^'^"^ ''^'^ "^""^^"^ ^^^^ 

CONCLUSIONS OF THE STUDY 

Effort must be made to consider what are regarded as adequate levels of 
writing ability. For example, what is the effect of correct subject-ve?b 
agreement on a student's overall writing ability? As has been mentioned 
of the students in the lowest holistic score grolp have denSmstrated t^^^^^^^ 

rorT''HUter'"does'thr5''' subjects and Wbs agree, and 'tM s wi\h feJ 
!^J!Iro u?!* • ^''^s that they are good or even moderately good 
yrii 7kL*"'? """l* considered is that students of low ability might knew 
that they do not know how to deal with the more difficult or comoleJ 
structures. Therefore, by avoiding, for example, the use o? nreoositiSnal 

STi's'smi" ''''' can^;n^l;ue"ith mfjrre ? 

ifl this skill. Yet the same group of students can and do resDond with 
differing levels of skill on multiple-choice items requirtSg an identifica? on 
of the correct structure, perhaps from a list of controlled difficulty This 
aMmT" """"^ adgflUaifi levVls of wHU 

The types of writing range from that generated by students who use simDle 
structures incorrectly or correctly to that of students who use cSmSl ex 
structures incorrectly or correctly. Where "adequate writing" falls alonTthe 
cont nuum between these extremes is not a Question that can be e?silv 

Zlnl'li' J "^^^^ """^^ recognized. Assessing writing skills bj 

means of objective measures yields one set of results while direct assessment 
through production writing may yield another. That each has itsldvaJtagIs a!d 
disadvantages suggests that the exclusive use of one or the other mavfefdfn 
Misinformation regarding the level of students' writing skills 
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Overall, then, the data provided herein indicate the followtng: 

(1) Results of the SSAT-I agre« reasonably closely with those from 

J!? "^''^ f^^^^Vi^" when the students have 

fairly high levels of overall writing skill. 

(2) For below-average students, scores representing the same skills tend 
to be lower on the SSAT-I than on the analytically scored papers. 

*° penalize students with lower levels of 
overall writing skill. Moreover, the SSAT multiple-choice format may 
even lead to a misdiagnosis, particularly in verb usage, of these 
students' major problems. " 

(4) Such diagnostic inaccuracies and/or insufficiencies, and the 
deficiency label placed or schools when SSAT-I scores in a skill area 

tI^ "^''m"* ""^^ *o incorrectly focused 

l!f.!in«t P^^i'^l^ '•""It can be to harm rather than help the 

student, and to misinform the instructor and/or the curriculum 
designer. Quite simply, these results appear (to us) to call into 
question the usefulness of the diagnostic information arising from 
SSAT-I writing results, at least at grade 10. 

(5) Using the writing portion of the SSAT-I as part of the criteria for 
labeling schools deficient is at best debatable because the SSAT 
appears to be a questionable indicator of the actual writing ability 
of low-level students. Further, because of the nature of writing 

r multiple-choice format writing portion d^es n?t 

skills adequate ass?? .npnt for the measurement of writing 
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APPENDIX A 
SSAT-I WRITING SKILLS ASSESSED IN 1987 
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SSAT-I WRITING SKILLS ASSESSED IN 1987 



$t^P(Jar<j/skiii QrMsrisiR Skin statpn.f>nt 

Sentence Co mposition 

* 10 V5 Make subjects and verbs agree. 

* 11 V5 Use appropriate forms of common irregular 

verbs. 

Organization 

^ 1^ V5 Generate headings for groups of words or 

phrases. 

^ 20 4/5 Organize information related to a single 

topic. 

Spelling. Punctuation. r aDitalizatini^ 

^ ^® VIO Spell words needed in writing through grade 

ten. 

^ V5 Apply generalizations for adding common 

suffixes. 

" V5 Use the comma to set off proper names in 

direct address. 

" V5 Use apostrophe and "s" for possessive nouns 

not ending in "s." 

^ ^* V5 Capitalize appropriate words in titles. 

^ V5 Capitalize proper nouns and proper 

adjectives. 
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APPENDIX B 
PRODUCTION WRITING ASSESSMENT 
PART I: HOLISTIC SCORING GUIDE 
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' HOLISTIC SCORING GUIDE 



4 P9INT 

An excellent paper has all or mo*; of the following: 

- a clear focus which is appropriate to the prompt 

- good organization including an introduction, details, and a conclusion 

" patterJ^* sentence structure, appropriate to the topic and varied in 

- specific and vivid vocabulary that shows insight into the topic 

- few or no errors in conventions of standard American English 

3 POINT 

A good paper has all or most of the following: 
" inS^"^!!?!?*' l! ^lear and appropriate to the prompt 

' lilt which may. however, lack a clear introduction, suffi- 

cient details, or a conclusion 

- effective and appropriate sentence structure in most sentences 

- appropriate vocabulary that shows some insight into the topic 

- some errors in the conventions of standard American English 

2 POINT 

An adequate paper has all or most of the following: 

- a focus which is appropriate but in many parts unclear 

- unclear organization in much of the composition 

" ft?S3iiJ[s sentence structure which may contain awkward con- 

- many errors in the conventions of standard American English 

1 POINT 

An Inadequate paper has all or most of the following: 

- a focus which is inappropriate and generally unclear 

- no recognizable organization 

" J;j!l!''I!!!*5 sentence structure that detracts from the meaning 

- vague or dull vocabulary 

' Eiglis""^""' ^" ^''^ °^ conventions of standard American 

0 POINT 

This paper is blank, illegible, written in a foreign language, has insuffi- 
cient amount of writing to evaluate, or is totally inapprVpViate to the 
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APPENDIX B 

PRODUCTION WRITING ASSESSMENT 

PART II: EXAMPLES OF HOLISTICALLY SCORED STUDENTS' ESSAYS 
FROM THE 1986 PRODUCTION WRITING TEST 
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1986 PRODUCTION WRITING TEST 



PROMPT: Consider- aU o f th e t e ach eps you-Mve ha<l-ln -school. 
Identify one that you consider unforgettable. Describe that 
person so the reader will know why he or she Is so metnorable to 
you. Include Incidents or memories to show why that teacher Is 
unforgettable. 

HOLISTIC SCORE: 4 

PROMPT NUMBER: 13 
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<^<y> ^ ^gy^. 

U/9f^ yf^fyj yy^ /^rV/^ yrry yAff.y jf^y/:^/' 



i.^^y^ A^^j^il^^. f'^y .^^J yyi^' 



// ay>A,y Ay^ /^/^ W>- ^^^^ 21 



1986 PRODUCTION WRITING TEST 




^^^^ V^J- 



^A<L^ ^/^^y y/y.^r^ yfy^ 

^^Wid^ ^j^y^y^^ Jy^ ^^^^^ 




//V y^/^./ y^/y}yCl^y-;rrp^ //^^ /../.^ ^ 
'/ yCy / \ ^ 



Prompt Nuir^er: 



/I 



1986 PRODUCTION VRITING TEST 



HOLISTIC 8CCRE1 A 



\% xTv^ J^^kavXVn <Y><x^ K^^JU ^ VAr^.. 



o,^<tsC? <0ivs>. voa.*^ V\vA. o.^j.\ w)W V^c\ 



er|c — . 



1986 PRODUCTION WRITING TEST 



WV. r^o. rxVcWx y . <kV^ ACA Wc 

^^^^ ^^^^ rvxW<LV ^^jSiV Vnq^^C Wov a ^ 

^^^^^ CQ^^cx-x <^voC.^Cv.^.r. n^^^ i^Vv^ '. 

^•^A ^^^v'^^^g -^'^^^ o^o^A c.-XSL cko>,v-i^ 3, 

^ Ck(.A^ OiU- <^ ClX.O^JCL NQ<i=^e€x,^vSL-^ oP VW 

t 
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Prompt Number: \^ 

24 



\ 



1986 PRODUCTION URITIN6 TEST 
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O O ' Prompt Number: 



CO 



1986 PRODUCTION WITIN6 TEST 



HOLISTIC SCOREi 3 



.MWTNUHBER, |3 



mermrrj^e. frr>t% mr.\ - 

hirn Ot^ 0 4fr,r^>/'^ irymn^ hf . .ot!r>ii.i 
Ww>/>icrvir'^-i-io.>^^ Kv. (iOniiW nssiigk-t (i*s 

Open C\r,(J^'K a> r.\ir. ,^^Prt i /f^-it^ 

A<> ^Kv^ 'R-s fyi ^-tf^^LiO 11^ 'r( p'lhij 
ER?c (^t <y\^ <yime vrxj r. uK^iP i 



ItU PRODUCTION URITIN6 TtST 

^i>^4i'-p.5^ rv^-iWi-.^ •Q'« VMiyl n(im/jS 



An 5;p/'. n ^AYi^ik. i<'nrtnp6' a\AVTtJirAA 

_ _ - i»Av# TTT _ ^ ^. . nJ 



v^r\a ir> /TinA/l iQg, lot -C l^mf^ L-n 
Pj-y^ (Sn.XrPm *s/Y-f' narpf 



or! Pronpt Nuitber; 



U86 PRODUCTION URITINS TEST 



0 0 



Prompt Number: 



1986 PRODUCTION URITIN6 TEST 



HOLISTIC SCORE f 3 



moHFT nuhber: 



JA 




4. ^ISaAji^. (Ls^jIA lyi^/i. i.A>4.. iJL^x^ jJvfi A^.it^. — 



1986 PRODUCTION WRITING TEST 



ff 



tr 
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J(] "0 " "'"^''P^ Number: / 



i 1986 PRODUCTION WRITING TEST 



9' 



A ^ull fo^<.L, k^^,^.. iJu ^o Ml ^fALJ Ur^l- 



AruL A// l-Lc^ 



11 A^tb-tiL ^fC^^^ l^f^ 



ij;AV I; 
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Prompt Nuflter: 



1986 PRODUCTION WRITING TEST 



HOLISTIC 6CORE1 e 



PROMPT WUMBEKl. 1^ 
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g^O 'v/i^ — ^ n^^^ ^ v-^ ^w^iL ^pT-^ 

^P^jg. ^0? ^JLif .j^.^:^. 



iL^4. ^ .DftJL 



1986 PRODUCTION KRITIN6 TEST 




33 Prompt Number: 



1986 PRODUCTION WRITING TEST 



HOLISTIC 8CORE1 2 

nam NumERs 



{^yJGLy-^^s Ti/-L^ y^^Aj^ 

ERIC /^^^ y-T^x^T^^^.^ / ^^^^ 



1986 PRODUCTION WRITING TEST 



Vl/O^-i ^^/-y yr? 3>r>y^ 








Proapt Nuoier: X3 



198C raOOUCTlON NRITtW TEST 



£:>^^r^,jr^ .sts^ y^^y^ 



^ Prompt Number: 13^ 

.1o 



1986 PRODUCTION URITIN6 TEST 
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HOLISTIC SCORES I 



PROMPT NUMBERS 



^/i yyJ/>^J 





21. 



1986 PROOUaiON WRITING TEST 



HOLISTIC SCORES 1 



fRONPT NUMBERS 




'^^^^ ^^^f^^ ,v^<s»g.^^ 
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^^^^^^ ^Z*\^f 




1986 PRODUCTION WRITING TEST 



ERIC 




f*^^^^^'^^ ^ ' ^-^^^^ . 4^^^.^^ ^ ^ ^ 
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APPENDIX C 

RESULTS OF ANALYTICAL SCORING 

COMPARISONS OF RESULTS FOR SKILLS 
WHERE GREATEST DATA DIFFERENCES OCCUR 
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COMPARISONS OF RESULTS FOR SKILLS 
WHtR£, GREATEST DATA DIFFERENCES OCCUR 



A 10 - MAKF SUBJCCTS AND yppR^ ftcpfp 

IJlL*'^^" ^*^P*c^®^f" ^^'^^ 38% of the students in holistic writing 
bS? in^Tf.ii"^' '"I' ^l* Sroup 5-6. and 93% of group 7-8^ 

But in actual writing performance, students in all g" ps "mastered" this 
skill much more frequently than one might expect from the SSAT scores. 

II!Sn3''^^^5fit?^7fi■T,^''f^*'®®" analytical results and the SSAT scores occurs 
fI?;Mi¥''®Ci?/I*\i'*^Jfi^f '"'■^ 2-3. Only 33% of these students passed 
this SSAT skill by identifying the correct form of subject or verb in at least 

sk?n^n't'hV\1S'nf'^'" ^^V"^,' "° errorsTt all in'?h?s 

?S H-on^J^^h^ 5KtrdVrsVa^ti^n°g rsVeroTtS7s°7kiiriJ 

jrihe's^AM^^^^^^ ' ''''^''^^^ P°^"'l"moJS 

Of the students whose holistic writing score is 4, 60% passed the subiect-verb 
agreement skill on the SSAT. However, in the fifteen essays studied f?om this 
2i;:^;Hfn*tJ„*^ "° "7°" ^" subject-ve-b agreement. Moreover, no stud?It had 
more than ten percent errors. Thc.«s. there is a difference of forty o-rcentaae 
poi.:ts between the analytical scores and the 5SAT results for this skill ^ 

J!??"^ ^ff. ''e«»a^"^"g score groups. 5-6 and 7-8. all students "mastered" 
J?\MNlIt:? pis^SSj "inr^''^ - thf ^^^rmer^^^^l?. 

gicfeS^^:^^^^ ^T't^at^U^^f^^^^^^^^^ o?W^{esn:Sl 

eSl C^t'Z''^!. J^Ml^r'"^ k''^. P°°^ Pt.-formance on produJ tSS 

Items, comparing the subject-verb forms contained in the SSAT items with th* 

\V«r5/°""' *'"PP"*^ the essay writers might be helpful * Along thi 
five SSAT items were .wo sentences having plural subjects need I no JimolP 

P°\the.^e. students had to select the correct subScrfom 
one and the correct verb form for the other. A third item was a sentence^ with 
;««i??id*L'"''^i''\^°"?^ * prepositional phrase. For this! stuSpnts ilJe 

s3JcT^was /«2o«^^^^^^ t'KHfJ"^"^''/'^''- ^" '^^'"^^"^"g two items. 5nl 
suDjeci was compound, the other was plural, preceded by a comoound adiprtivn 

Students were required to select similar 'answers, both pSl verb oSfJse' 
?«:L'^;lS„;\r^lll7.''''^^ P"* participle fori of an irregulS? 

(even^though the skill does not specify students' knowledge of irregular yllb 

JSrS^^rimn'/r'Vn'*;?^"^"?^'"*"^ J^^P^® Pl"''*! ^"Wects and simple plural 
verbs similar to the first two aforementioned SS../ items (i e blural 
subjects needing simple plural verbs). However, only c few essays confaiJed 
tne compound subject/plural verb form, and only in the 7-8 group d^deJsSvs 
contain a subject and verb interrupted by a prepositional ohras- tho 
frequently used forms consisted of a' singu'lar o^p^ural ^%l? su^^^^^^^^ 

rl!«Illf^"ii^? security, actual SSAT-I test items are not included in this 
report. Their substance, however, is connoted for the sake of clarity 
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ofTl^lJI H^^lll VJ£^^ i®'"''- Verb phrases, when employed, usually consisted 
Of was-were or has -have and a present or past participle. 

This accounts for the high levels of performance across all holistic score 
student-generated subject-verb agreement. The frequency of simple 
'"^gests that these forms are the most familiar to 
! this is the probable reason they are used more frequently than 

?f5fr;n"^T^J*T ^PP"*"^^ t**" 0^ three times (Jf the 

nvej on the SSAT. Even if students answered these two or three correctly 
mere guessing at the remaining items might not have been enough to correctly 
answer the four or five items required to demonstrate mastery of the skill. 

A 11 - USE APPROPRIATC FfiBM^ QF r.OMMON TRRFGULAR VFPRf; 

On the SSAT. students demonstrated this skill by selecting one of four verb 

89%. and 97% of holistic writing score groups 2-3. 4 5-g and i-k 
respectively. Comparing SSAT results to students' actual writina' oractices 
IfZlJ^""!" r!:^^!j^"J' difference exists between the scVes amSSg 
students in the three highest holistic score groups, again a difference ii 
found in the 2-3 group All papers studied for this report employed son^ forms 
SLn'^^tS^*'' twenty-five essays from the lo£ hoi istiT score 

group. 52% were free of this error, and another 24% had less than the maximum 
twenty percent error rate allowed by the SSAT. Thus. 76% of these stJdeStJ 
demonstraced this skill successfully in actual writing, one third a^re thaS oS 
the SSAT. In the next higher writing score group. 4. 53% of the oaoers 
f ^''^'•^ h^^* "P to%ixteen'pe?«nt errors.'^?Sr a 

-MsliX^ tMt5?ll''iiSJr'';S °! imoprUte irrSguUr verbs aid tJus 
passing this skill; whereas. 75% passed this portion of the SSAT. 

^?nI«ntV^n^^^^ ^'•'^"lai: verbs featured in the SSAT items were used by 
students in their essays, but many other irregular verbs also aooeared Most 

a'rs^ii'Vs fomsTf^;^^^^^^^^ "felt/ a^'S"^hy;^ 

as wen as forms of be. "go." and "do.^ The essays in the 7-8 qrouo were 



S 58 - SPFI L WORDS NFFHFn TM |tRITTNG THPniiqH qpflpp |n 

In this skill also, the 2-3 papers contained fewer overall errors than the 

ra?I orthirt'v^^V^*!!? r^^r^^./iHpS"* The SSAT allows aTximum erJJr 
rate of thirty percent for this skill, but in the analytical scorina the 
maximum was seven percent for students in the 2-3 group This Jas done becaSi 

c5??i; l%'"Thl't\'llpr/fir^^ ^"""^^ ^''^'^'''y thinkThercan spe 1 

correctly. The smaller allowance for errors somewhat compensates for the 

sJilleTwo^dV^^^^^^^ •'J"^' f ''fry^cV^'^'^ identifying ten coVr'ectlJ 

sS^ ino or FlA^^ aS !s\"V^^^^ V'.* '■^^^^ ^o*" Assessment of 

stSiin?'; «1 -^A' ?^ supplying correctly spelled words from the 

student s own "master list." Even with this lower error rate applied to their 

nusTbL n?^Pd%'H^VtV^^^^ 9TP "P""<'" this skflll However 'u 

must be noted that thirty-nine percent of the 209 words missoelled bv these 
students are FLAS words from grades three, five, eight! aid eleven oXly tJo 
Ji^ors"'5n'F!lAS°w5?dr' '''' s^ven 'students haVfi've or'm5?e 
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G ei ■ APPLY GENERAL I7ATT0NS FOR AonTNR ^nm)^ ^fypp^fc 

This skill requires students to identify the correct spelling of a word formed 
by add ng a given suffix to a given word. For example, a typical item mighrbe 
the following: If the suffix is added to the "urd liaaS. which ansSer Vs 
the correct spelling of the new word? The options m™ include spelling 
variations such as "happly.- "happyly.- happilly.- as well as. "happily " 

?IffiJ2f u.?lL"''thV'%«?*^' "^''^ studied for correct spellings of words with 
^SIuUp5^n ?n ^! i^J error rate of twenty percent as the criterion 
resulted in all students "passing" with lOOX accuracy. This included the 
lowest group, of which only 6SX passed the same skill on the SSAT Among 
essays in writing score gmup 4. analytical scores for this skill werl 
nla^tl^l ?^''"^"V?f "^9"®'' 0" SSAT, Essays in writing score 

thSS for the SSAT " ' '"espectively. ten and two percentage points more 

55!V«?llllf.ti!^^ ^aaP students in the low holistic score group were 

Sr^fJj ^"<'^"ting plurals (19 essays) and present or 

tJrL'^tw respectively). Each form was misspelled 

SeoSentlT Mn a' ^^h' '^Z f^"" ^"""^^ most 

InfJ In ^1 h ^' t' w^th misspellings occurring 

only in the -ly form. Several other suffixes appeared without errors in onl? 
one or two essays each, such as -est. -ful. -ous. and -en 



I 75 - CAPITAI IZE PROPFR wniiM s AND PROPFR AfllFr^T Typ^ 

On this skill, the group of students with holistic scores of 2-3 oerformed 
Irf^jf writing than on the SSAT. although the difference befSeenThe 
iJSnJ! «f L '7*'^®'' "l®?, skill. However, for all other 

Tclrll ^^"^'entS' analytical scores here were lower than were the SSAT 



Recent SSAT-I items used to measure this skill contained context clues which 

SJe^Tiie^tif!''?!:!? «of?In' ''''''''' ^" items stuSSit 

were to identify that capitalization was needed in the following cases: 

^ -ilSlilSr;**^ * ^''^ contained the word 

^' bSsine^;'"^ * corporation which the stem implied to be a place of 

^* JnnJ!;!!i'°''?*K*'!! °^ * not likely to be easily 

adjectives *'h.^^^^^ "f"^"^" P^-^^eded by two commoJI 

Mjeciives because the first two words were capped; 

4, The name of a mythological figure referenced by the word "goddess" in 

5. The name of a language, for which the test stem contained the name of 
another language as well as the word "language." 

J?oS2"\di'rctVv'S' "SJ!t'n!lin2l' ""^ * P/^P^*" but few used 

proper aajectlves. Most proper nouns were names of persons with names nf 

places occurring next most often. For example, in writing score grSrVS 
?Jn<;rliLf?^ "f^" °" ^''^^ f^^" "^''^ twenty pe?Lntage poTts ?roi 
ISU^I ttlJ^IsSI oTp^^s.^^^^ '''''' ^PP"^ 
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BUREAU RESPONSE 
TO REPORT* 



♦The following section is a response initiated and developed by the Bureau of 
Education program staff and is not an official part of the report itself. 
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December 16, 1987 



RESPONSE TO THE FEBRUARY 1987 TENTH GRADE PRODUCTION WRITING ASSESSMENT 



Since there are implications for the Department of Education, the 
study should be transmitted to the Assessment. Testing, and 
Evaluation Section of the Florida Department of Education for its 
considersation. 

This recommendation has been reviewed by Mrs. Zelda Glazer. Dr. Charles 



REPORT 






hrank de Varona 
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ThsSdnolBo^ 

nondncrinHMtUNi ni sducitionri uoyiBii/wtivilJis an anployinent 
nl strivBi iffiiiiHlMly to pravkfe 6quil {qtoduntyforwairequiiBd 
by: 

Title VI of 1l» Qvil Krfm Act of 1964- praMlRts.fiaamnition 
on the bM ol racs, color, iriigioiv ^ iMoral origin. 



Titte VII of tte Ovii Riniiti Act of 1 9di,m mendBd ■ prohifarts 
dhuiniiwtion in snplcynnnt cn Iho bw of m, color, religion^ 
$89^ or mtioml oripHi 

TitleJX of the EdCTtion Amendrnrm of 1972 - prohibits 
dncrinvrallon on the bM of Kx. 

AfB DiKrininition Act of 1967, « msndMl - orohibi^ 
cnmmtxm on thi bM of botMBen 40 ana 7L 

SKtion m of the fWMiliti^ 



Fioride Eduertonrf B|uit yArt' proWbi ti dhommtiaion the 
beiii of race, m, mtioM origin^ nMritri stilui or hendkap 
igMtt I fiikhnt or MployM. 

VetBin se pravjded re fliuloynMit rights in nordnse with P.L 
93-W (FedepiO wdSwaon a i.07, Horidi Statutes, which also 
stipulatas cetegiMical iNofeieiM for mirioynMiL 
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